Skip to content

fix(bigquery-driver): forward stream errors and propagate consumer cancellation via pipeline#10877

Open
tlangton3 wants to merge 1 commit into
cube-js:masterfrom
tlangton3:cube/fix-stream-mode-unhandled-rejections
Open

fix(bigquery-driver): forward stream errors and propagate consumer cancellation via pipeline#10877
tlangton3 wants to merge 1 commit into
cube-js:masterfrom
tlangton3:cube/fix-stream-mode-unhandled-rejections

Conversation

@tlangton3
Copy link
Copy Markdown
Contributor

@tlangton3 tlangton3 commented May 13, 2026

Summary

BigQueryDriver.stream wires its returned rowStream to the underlying @google-cloud/bigquery source stream via stream.pipe(rowStream). By design, Node's pipe() forwards data/end events but not error events. When BigQuery returns an HTTP error mid-stream (e.g. a type-coercion rejection like No matching signature for operator = for argument types: TIMESTAMP, DATE), the source stream's 'error' event has no listener — Node's default 'error' handler fires (processTicksAndRejections) and the cubejs-server process exits with code 1. Every BI session connected to that pod is torn down with server closed the connection unexpectedly.

The non-streaming path (driver.query) is unaffected because the rejection propagates through await and is caught by the cube native bridge's top-level try/catch. The streaming path returns the rowStream synchronously, so await resolves before the HTTP call fires — the bridge has no chance to catch the rejection.

Fixes #10875.

Changes

Switches BigQueryDriver.stream from stream.pipe(rowStream) to stream.pipeline(stream, rowStream, () => {}) from node:stream. This fixes both observed defects:

  1. Source error forwardingpipeline auto-destroys rowStream with the underlying error → bridge's rowStream.on('error', ...) handler fires → wire layer emits structured Postgres ErrorResponse (XX000) carrying the verbatim BigQuery message.
  2. Consumer-side cancellation — destruction of rowStream (client cancellation, severed BI session) now destroys the source too, preventing the driver from paging results into the void after the consumer has gone away.

Implementation Details

  • pipeline() is in Node's stable API since v10 (callback form). Cube's Node engine constraint is well clear of this.
  • Callback is intentionally a no-op: pipeline already destroys rowStream with the error, and the callback exists solely to satisfy the signature and prevent an unhandled rejection inside pipeline itself.
  • Promise-based stream/promises.pipeline is the wrong choice here — awaiting it would block returning rowStream until the entire query completes, buffering the whole result set in memory. Callback form keeps the existing synchronous-return contract.

Testing

Unit tests (new — packages/cubejs-bigquery-driver/test/BigQueryDriverStreamError.test.ts)

Two synthetic-source tests using PassThrough streams (no real BigQuery needed):

  • forwards source-stream errors to the returned rowStream — emits an error on a mock source and asserts rowStream receives it.
  • propagates rowStream destruction back to the source stream — calls rowStream.destroy() and asserts the mock source's destroyed flag is set.

Both tests time out when the fix is reverted to bare stream.pipe() (proves they catch the regression).

End-to-end verification against real BigQuery

Verified via the cube SQL API + psql (cube v1.6.46 with patched BigQueryDriver.js overlaid into a running container):

Path Before After
100,000-row streaming success works works (no regression)
BigQuery TIMESTAMP=DATE error container exits 1, server closed the connection unexpectedly ERROR: XX000: Database Execution Error: No matching signature for operator = ... Signature: T1 = T1 ...; container alive

Wire-level behaviour is now identical to the non-streaming SQL API (same XX000 SQLSTATE, same pg.DatabaseError shape, same verbatim BigQuery message reaching the client).

Compatibility & risk

  • No behavioural change on the success path (verified by the 100k-row streaming check).
  • Failure path now surfaces a structured Postgres error where it previously crashed the process — strict improvement.
  • stream.pipeline is the textbook Node primitive for this exact wiring. No new dependencies.

…ncellation via pipeline

`BigQueryDriver.stream` was wiring its returned `rowStream` to the
underlying `@google-cloud/bigquery` source stream via `stream.pipe()`.
By design, Node's `pipe()` forwards `data` and `end` events but NOT
`error` events. So when BigQuery returned an HTTP error mid-stream
(e.g. type-coercion rejections like `No matching signature for
operator = for argument types: TIMESTAMP, DATE`), the source stream's
`'error'` event had no listener — `processTicksAndRejections` fired on
Node's tick queue and killed the cubejs-server process. Every BI
session connected to that pod was torn down, with `server closed the
connection unexpectedly` over the wire and the actual BigQuery error
visible only in the container's stderr.

The non-streaming path (`driver.query`) is unaffected because the
rejection propagates through `await` and is caught by the cube native
bridge's top-level `try/catch`. The streaming path returns the
`rowStream` synchronously, so `await` resolves before the HTTP call
fires — the bridge has no chance to catch the rejection.

Switching to `stream.pipeline` fixes both observed defects:
  (a) source-stream errors are auto-forwarded by destroying `rowStream`
      with the same error, so the bridge's `rowStream.on('error', ...)`
      handler fires and the wire layer emits a structured Postgres
      `ErrorResponse` (SQLSTATE XX000) carrying the verbatim BigQuery
      message;
  (b) consumer-side destruction of `rowStream` (client cancellation,
      severed BI session) now destroys the source too, preventing the
      driver from paging results into the void after the consumer has
      gone away.

Verified end-to-end against real BigQuery via the SQL API + psql:
both the success path (100,000 rows streamed cleanly) and the failure
path (BigQuery TIMESTAMP=DATE rejection surfaced as XX000 with the
verbatim message) now behave correctly; the cube container stays up.

Adds two synthetic-source unit tests (`BigQueryDriverStreamError.test`)
verifying both directions of the lifecycle propagation. Without the
fix, both tests time out (proves they catch the regression).

Fixes cube-js#10875.
@github-actions github-actions Bot added driver:bigquery Issues related to the BigQuery driver rust Pull requests that update Rust code javascript Pull requests that update Javascript code data source driver pr:community Contribution from Cube.js community members. labels May 13, 2026
@tlangton3 tlangton3 force-pushed the cube/fix-stream-mode-unhandled-rejections branch from b80dbe0 to 158565e Compare May 14, 2026 07:27
@tlangton3 tlangton3 changed the title fix: stream-mode unhandled-rejection vectors (cube-js/cube#10875) fix(bigquery-driver): forward stream errors and propagate consumer cancellation via pipeline May 14, 2026
@tlangton3
Copy link
Copy Markdown
Contributor Author

Force-pushed to drop the .unwrap() defensive hardening that was originally on this branch as a second commit. That work has moved to its own PR (#10882) for clean scope — it's a separate defensive change in the Rust native bridge, structurally unrelated to this BigQuery driver fix. This PR's head is now 158565eb6 and addresses #10875 on its own.

@tlangton3 tlangton3 marked this pull request as ready for review May 14, 2026 08:48
@tlangton3 tlangton3 requested a review from a team as a code owner May 14, 2026 08:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data source driver driver:bigquery Issues related to the BigQuery driver javascript Pull requests that update Javascript code pr:community Contribution from Cube.js community members. rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming SQL API: BigQuery driver does not forward stream errors, killing pod on any mid-stream BQ failure

1 participant